Electronic apparatus and controlling method thereof

ABSTRACT

An electronic apparatus may include a memory that stores first information regarding a plurality of first artificial intelligence models trained to perform image processing differently from each other and second information regarding a second artificial intelligence model trained to identify a type of an image by predicting a processing result of the image by each of the plurality of first artificial intelligence models. The electronic apparatus may further include a processor configured to identify a type of an input image by inputting the input image to the second artificial intelligence model stored in the memory, and process the input image by inputting the input image to one of the plurality of first intelligence models stored in the memory based on the identified type.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority from Korean Patent Application No.10-2019-0150746, filed in the Korean Intellectual Property Office onNov. 21, 2019, the disclosure of which is incorporated herein byreference.

BACKGROUND 1. Field

Aspects of exemplary embodiments of the present disclosure relate to anelectronic apparatus and a controlling method thereof and moreparticularly, to an electronic apparatus that processes an input imageusing an artificial intelligence model and a controlling method thereof.

2. Description of the Related Art

Recently, an electronic apparatus may perform image processing using aneural network technology which is represented by deep learning that hasbeen developed rapidly. For example, the neural network technology maybe applied to segmentation, super-resolution, HDR, etc.

Such a neural network technology may be implemented in the form of anembedded system as shown in FIG. 1A in consideration of delay time.However, because of a limited computing resource, neural networkalgorithms may be optimized, but they do not reflect all thecharacteristics of inputs. Specifically, if a single neural networkmodel is applied to all inputs, desired results may not be produced forall inputs. In other words, according to the characteristics of an inputimage, side effects may be generated together with a desired imagequality improvement(s).

For example, as illustrated in FIG. 1A, in the case of a neuralnetwork-based super resolution model trained as a general image, aresult without side effects may be obtained in the general image.

However, as illustrated in FIG. 1C, various side effects (e.g., jaggingor blurring) may be generated in an image with text.

Due to such side effects, applying the same neural network model to allinputs is a problem in neural network technology.

In order to resolve such a problem, after the neural network processing,there may be a method of adding a module for sensing and improving sideeffects. Alternatively, there may be a method of storing neural networkmodel information to be used for various input images and providing thesame to an embedded system.

However, in the former “sensing” case, the side effects of an image(e.g., after upscaling) should be sensed and eliminated, but the methodis not appropriate in an environment where computing resources arelimited. In the latter “storing” case, there is a problem that there isno way of processing when there is no information in the input image.

SUMMARY

The present disclosure is provided in accordance with theabove-described needs, and aims at providing an electronic apparatus forimproving the processing performance of an input image by changing aprocessing method according to the type of the input image andcontrolling method thereof.

An electronic apparatus according to an embodiment of the disclosure mayinclude a memory in which information regarding a plurality of firstartificial intelligence model trained to perform image processingdifferently each other and information regarding a second artificialintelligence model trained to identify a type of an image by predictinga processing result of the image by each of the plurality of firstartificial intelligence models is stored and a processor configured tobe connected to the memory and to control the electronic apparatus, andthe processor is configured to identify a type of an input image byinputting the input image to the second artificial intelligence model,and process the input image by inputting the input image to one of theplurality of first intelligence models based on the identified type.

According to an embodiment of the disclosure, a controlling method of anelectronic apparatus may include identifying a type of an input image byinputting the input image to a second artificial intelligence modeltrained to identify a type of an image by predicting an image processingresult by each of a plurality of first artificial intelligence modelstrained to perform image processing differently each other andprocessing the input image by inputting the input image to one of theplurality of first artificial intelligence models based on theidentified type.

According to an embodiment of the disclosure, an electronic apparatusmay include a memory that stores first information regarding a pluralityof first artificial intelligence models trained to perform imageprocessing differently from each other and second information regardinga second artificial intelligence model trained to identify a type of animage by predicting a processing result of the image by each of theplurality of first artificial intelligence models, and a processorconnected to the memory and configured to: identify a type of an inputimage by inputting the input image to the second artificial intelligencemodel, and process the input image by inputting the input image to oneof the plurality of first intelligence models based on the identifiedtype.

The processor may be further configured to down-scale a resolution ofthe input image and identify a type of the input image by inputting thedown-scaled input image to the second artificial intelligence model.

The processor may be further configured to, based on a size of theoriginal image being equal to or greater than a predetermined size,down-scale the original image to obtain the input image, and identify atype of the original image by inputting the down-scaled input image tothe second artificial intelligence model.

The second artificial intelligence model may be an artificialintelligence model obtained by training a relationship between aplurality of sample images and a type corresponding to each sample imagethrough an artificial intelligence algorithm.

Each of the plurality of first artificial intelligence models may be anartificial intelligence model obtained by training a relationshipbetween a sample image of a type corresponding to each of the pluralityof first artificial intelligence models from among the plurality ofsample images and the original image corresponding to the sample imagethrough an artificial intelligence algorithm.

The type corresponding to each sample image may be obtained byprocessing each of the plurality of sample images by inputting each ofthe plurality of sample images to one of the plurality of firstartificial intelligence models, and obtained based on a type of a sideeffect of each of the processed plurality of sample images.

The type of a side effect may include at least one of none, jagging,blurring, aliasing, or noise boosting.

The processing the input image may include at least one of up-scalingprocessing, noise removing processing or detail enhancement processing.

The processor may be further configured to obtain a weighted valueregarding a plurality of types related to the input image by inputtingthe input image to the second artificial intelligence model, andidentify a type having a largest weighted value from among the pluralityof types as a type of the input image.

The processor may be further configured to, based on at least one ofweighted values regarding the plurality of types being equal to orgreater than a threshold value, store the input image and the at leastone weighted value in the memory.

The processor may include a central processing unit (CPU) and a NeuralProcessing Unit (NPU) which operate based on an Operating System,wherein the CPU is configured to identify the type of the input image byinputting the input image to the second artificial intelligence model,and wherein the NPU is configured to process the input image byinputting the input image to one of the plurality of first artificialintelligence models based on the identified type.

According to an embodiment, a method of controlling an electronicapparatus may comprise: identifying a type of an input image byinputting the input image to a second artificial intelligence modeltrained to identify a type of an image by predicting an image processingresult by each of a plurality of first artificial intelligence modelstrained to perform image processing differently from each other; andprocessing the input image by inputting the input image to one of theplurality of first artificial intelligence models based on theidentified type.

According to an embodiment, a non-transitory computer readable mediumstoring computer program code, which, when executed by a processor, maycause the processor to perform a method of a method that includes:identifying a type of an input image by inputting the input image to asecond artificial intelligence model trained to identify a type of animage by predicting an image processing result by each of a plurality offirst artificial intelligence models trained to perform image processingdifferently from each other; and processing the input image by inputtingthe input image to one of the plurality of first artificial intelligencemodels based on the identified type.

According to an embodiment, a computer-implemented method of training aneural network for image processing may comprise: collecting a set ofdigital sample images from a database; inputting the collected set ofdigital facial images into a plurality of first neural network models,so as to obtain a plurality of outputs, wherein the plurality of firstneural network models are trained to perform image processingdifferently from each other; training, in a second neural network model,relationships between the set of digital sample images and a side effecttype corresponding to each digital sample image using the obtainedplurality of outputs, wherein the side effect type corresponds to a sideeffect of processing by a respective one of the plurality of firstneural network models.

The computer-implemented method may further comprise: identifying a sideeffect type of an input digital image by inputting the input digitalimage to the second artificial intelligence model, and processing theinput digital image by inputting the input digital image to one of theplurality of first intelligence models that corresponds to theidentified defect type.

According to the various embodiments of the present disclosure, theelectronic apparatus may identify the type of an input image using anartificial intelligence model, and improve processing performance of theinput image by processing the input image using another artificialintelligence model corresponding to the identified type.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features, and advantages of certain exampleembodiments of the disclosure will be more apparent from the followingdescription taken in conjunction with the accompanying drawings, inwhich:

FIG. 1A is a view provided to explain a problem of a conventionaltechnology;

FIG. 1B is a view provided to explain a problem of a conventionaltechnology;

FIG. 1C is a view provided to explain a problem of a conventionaltechnology;

FIG. 2 is a block diagram illustrating configuration of an electronicapparatus according to an embodiment;

FIG. 3 is a view provided to explain an operation of a processoraccording to an embodiment;

FIG. 4A is a view provided to explain an operation of a processor basedon an image type according to an embodiment;

FIG. 4B is a view provided to explain an operation of a processor basedon an image type according to an embodiment;

FIG. 5A is a view provided to explain a training method of a pluralityof first artificial intelligence models and second artificialintelligence models according to an embodiment;

FIG. 5B is a view provided to explain a training method of a pluralityof first artificial intelligence models and second artificialintelligence models according to an embodiment;

FIG. 5C is a view provided to explain a training method of a pluralityof first artificial intelligence models and second artificialintelligence models according to an embodiment;

FIG. 6 is a view provided to explain real time computation according toan embodiment;

FIG. 7 is a view provided to explain an update of a plurality of firstartificial intelligence models and a second artificial intelligencemodel according to an embodiment; and

FIG. 8 is a flowchart provided to explain a controlling method of anelectronic apparatus according to an embodiment.

DETAILED DESCRIPTION

The exemplary embodiments of the present disclosure may be diverselymodified. Accordingly, specific exemplary embodiments are illustrated inthe drawings and are described in detail in the detailed description.However, it is to be understood that the present disclosure is notlimited to a specific exemplary embodiment, but includes allmodifications, equivalents, and substitutions without departing from thescope and spirit of the present disclosure. Also, well-known functionsor constructions are not described in detail since they would obscurethe disclosure with unnecessary detail.

Hereinafter, the present disclosure will be described in detail withreference to the accompanying drawings.

As for the terms used in the embodiments of the disclosure, generalterms that are currently used widely were selected in consideration ofthe functions described in the disclosure. However, the terms may varydepending on the intention of those skilled in the art who work in thepertinent field, previous court decisions or emergence of newtechnologies. Also, in particular cases, there may be terms that weredesignated by the applicant, and in such cases, the meaning of the termswill be described in detail in the relevant descriptions in thedisclosure. Thus, the terms used in the disclosure should be definedbased on the meaning of the terms and the overall content of thedisclosure, but not just based on the names of the terms.

In the present disclosure, the terms “include” and “comprise” designatethe presence of corresponding features (e.g., components, such as,numbers, functions, operations, parts, etc.), but do not exclude thepresence of additional features.

In the present disclosure, the term “at least one of A or/and B” shouldbe understood to mean any one of “A”, “B”, or “A and B.”

Expressions “first”, “second”, or the like, used in the disclosure mayindicate various components regardless of a sequence and/or importanceof the components, will be used only in order to distinguish onecomponent from the other components, and do not limit the correspondingcomponents.

Singular forms are intended to include plural forms unless the contextclearly indicates otherwise. In the present disclosure, the terms“include” and “comprise” designate the presence of features, numbers,steps, operations, components, elements, or a combination thereof thatare written in the present disclosure, but do not exclude the presenceor possibility of addition of one or more other features, numbers,steps, operations, components, elements, or a combination thereof.

In the present disclosure, the term “user” may refer to a person whouses an electronic apparatus or a device itself (e.g., an electronicdevice) using an electronic apparatus (e.g., an artificial intelligenceelectronic apparatus).

Hereinafter, an embodiment of the present disclosure will be describedin greater detail with reference to the accompanying drawings.

FIG. 2 is a block diagram illustrating configuration of an electronicapparatus 100 according to an embodiment.

The electronic apparatus 100 may be an apparatus for processing an inputimage. Referring to FIG. 2, the electronic apparatus 100 may include amemory 110 and a processor 120. The electronic apparatus may furthercomprise a display. The electronic apparatus may process the imagedirectly. The electronic apparatus may be an electronic device, such as,a television (TV), a desktop personal computer (PC), a notebook PC, adigital video disk (DVD) player, a smart phone, a tablet PC, a monitor,smart glasses, a smart watch, etc. Alternatively, the electronicapparatus 100 may be an apparatus without a display. The electronicapparatus 100 may further comprise a communication interface, which maybe controlled to transmit a processed image to a display device, suchas, a monitor, a set-top box (STB), a speaker, a computer main body,etc. However, the electronic apparatus 100 is not limited thereto, andmay be any apparatus capable of processing an input image. Thecommunication interface may be configured to perform communication witha server, a user terminal, etc. The performed communication may be via anetwork or a short range communication method that does not require anetwork.

However, the electronic apparatus 100 is not limited thereto, and may beimplemented in a form in which some components are excluded.

The memory 110 may store various artificial intelligence models. Forexample, the memory 110 may store information on a plurality of firstartificial intelligence models trained to perform image processingdifferently from each other and information on a second artificialintelligence model trained to identify a type of an image by predictingan image processing result regarding each of the plurality of firstintelligence models.

In the present disclosure, the plurality of first artificialintelligence models may be artificial intelligence models for performinga same type of image processing. In this regard, the first artificialintelligence models may be, for example, artificial intelligence modelsfor performing up-scaling of an image. However, the plurality of firstartificial intelligence models are artificial intelligence models forperforming image processing differently from each other and, forexample, the plurality of first artificial intelligence models may havedifferent methods for performing up-scaling of an image. In addition,the image processing method of the plurality of first artificialintelligence methods may be different according to the type of an image.For example, optimal image processing may be performed when an image ofthe first type is processed by one of the first artificial intelligencemodels, and optimal image processing may be performed when an image ofthe second type is processed by another one of the first artificialintelligence models.

The second artificial intelligence model may be an artificialintelligence model for identifying the type of an image (to beprocessed). The number of types of images may be preset, or the secondartificial intelligence model may be trained to determine an input imageas one of the preset number of types of images. For example, the secondartificial intelligence model may be implemented as a Mobilenet model oran Inception model that perform fast classification. In addition, thesecond artificial intelligence model may be implemented in a simplestructure in comparison with a plurality of first artificialintelligence models to improve the speed of performing computation.

The memory 110 may store a plurality of input images and an input imageprocessed by one of a plurality of first artificial intelligence models.In addition, the memory 110 may store an input training image to be usedto train at least one of the plurality of first artificial intelligencemodels or the second artificial intelligence model.

The memory 110 may be implemented as a non-volatile memory, a volatilememory, etc., but is not limited thereto. For example, a hard disk drive(HDD) or a solid state drive (SSD) may be used as the memory 110, andany element capable of storing electronic data can be used as the memory110.

The processor 120 may control the overall operations of the electronicapparatus 100. Specifically, the processor 120 may be connected to eachelement of the electronic apparatus 100 and control the overalloperations of the electronic apparatus 100. For example, the processor120 may be connected to the memory 110, a display, and a communicationinterface, etc. and control the operations of the components of theelectronic apparatus 100.

According to an embodiment, the processor 120 may be implemented as adigital signal processor (DSP), a microprocessor, or a time controller(TCON). However, the processor 120 is not limited thereto, and theprocessor 120 may include one or more of a central processing unit(CPU), a micro controller unit (MCU), a micro processing unit (MPU), acontroller, an application processor (AP), a communication processor(CP), or an Advanced RISC (Reduced Instruction Set Computing) Machine(ARM) processor, or may be defined as the corresponding term. Inaddition, the processor 120 may be implemented by a system-on-chip (SoC)or a large scale integration (LSI) in which a processing algorithm isembedded, or may be implemented in a field programmable gate array(FPGA) form.

The processor 120 may identify a type of an input image by inputting theinput image to the second artificial intelligence model. The type of theinput image may be preset. For example, the type of the input image maybe divided into five types from a first type to a fifth type, and theprocessor 120 may input the input image to the second artificialintelligence model to classify the input image as one of the first tofifth types. Here, the number of types of image may be the same as thenumber of the plurality of first artificial intelligence models.

The processor 120 may process the input image by inputting the inputimage to at least one of the plurality of first artificial intelligencemodels based on the identified type. For example, based on the type ofthe input image being the third type, the processor 120 may process theinput image by inputting the input image to a third one of the firstartificial intelligence models, the third one of the first artificialintelligence models corresponding to the third type from among theplurality of first artificial intelligence models.

The processor 120 may down-scale a resolution of an input image andthen, identify the type of the input image by inputting the down-scaledimage to the second artificial intelligence model. For example, theprocessor 120 may convert an input image having an Ultra-high-definition(UHD) resolution into an image having a standard definition (SD)resolution, and input the converted image to the second artificialintelligence model to identify the type of the input image. Through suchan operation, the computation time for identifying the capacity of thesecond artificial intelligence model and the type of the input image maybe reduced.

Based on the size of the input image being equal to or greater than apredetermined size, the processor 120 may down-scale the input image andinput the down-scaled image to the second artificial intelligence modelto identify the type of the input image. For example, based on the inputimage having a UHD resolution, the processor 120 may convert the inputimage to an image having an SD resolution and input the converted imageto the second artificial intelligence model to identify the type of theinput image. Based on the input image having an HD resolution, theprocessor 120 may not convert the resolution of the input image, andinput the input image to the second artificial intelligence model toidentify the type of the input image.

Meanwhile, the second artificial intelligence model may be an artificialintelligence model which is obtained by training the relationshipbetween a plurality of sample images and a type corresponding to eachsample image through an artificial intelligence algorithm. A pluralityof sample images may be obtained by down-scaling each of a plurality oforiginal images. In addition, the type corresponding to each sampleimage may be obtained by inputting each of the plurality of sampleimages to one of the plurality of first artificial intelligence models.For example, the type corresponding to each sample image may be obtainedby processing each of the plurality of sample images by inputting eachof the plurality of sample images to one of the plurality of firstartificial intelligence models, and obtaining the type corresponding toeach sample image based on the type of a side effect of each of theplurality of processed sample images. Here, the type of side effects mayinclude at least one of none, jagging, blurring, aliasing, or noiseboosting.

Meanwhile, each of the plurality of first artificial intelligence modelsmay be an artificial intelligence model which is obtained by trainingthe relationship between a simple image of a type corresponding to eachof the plurality of first artificial intelligence models from among aplurality of sample images and an original image corresponding to thesample image through an artificial intelligence algorithm. For example,one of the first artificial intelligence models may be an artificialintelligence model which is obtained by training the relationshipbetween a sample image of the first type and a corresponding originalimage through an artificial intelligence algorithm, and another one ofthe first artificial intelligence models may be an artificialintelligence model which is obtained by training the relationshipbetween a sample image of the second type and a corresponding originalimage through an artificial intelligence algorithm.

Meanwhile, the same type of image processing may include at least one ofup-scaling processing, noise removing processing or detail enhancementprocessing. For example, the processor 120 may use one of the pluralityof first artificial intelligence models to perform up-scaling processingwith respect to an input image. Here, the plurality of first artificialintelligence models are artificial intelligence models for performingup-scaling, but the image processing method may vary according to thetype of the image.

However, the present disclosure is not limited thereto, and the sametype of image processing may include different image processing.

Meanwhile, the processor 120 may obtain a weighted value regarding aplurality of types related to an input image by inputting the inputimage to the second artificial intelligence model, and identify the typehaving the largest weighted value from among the plurality of types asthe type of the input image. For example, the processor 120 may inputthe input image to the second artificial intelligence model and obtain aresult of the first type having the weighted value of 80, the secondtype having the weighted value of 20, and the third type having theweighted value of 40, and may identify the first type having the largestweighted value of 80 as the type of the input image. In the above, theweighted value is represented as a score, but this is only an example.For example, the weighted value may be represented as a probabilityvalue.

Based on at least one of one of the weighted values regarding theplurality of types being equal to or greater than a threshold value, theprocessor 120 may store the input image and the at least one weightedvalue in the memory 110. If it is assumed that the threshold value is 70in the above example, the processor 120 may store the input image andthe information that the input image is the first type in the memory110. Subsequently, the processor 120 may transmit the input image andthe information that the input image is the first type to a server. Theserver may update the plurality of first artificial intelligence modelsand the second artificial intelligence model based on the receivedinformation, which will be explained in detail later.

Meanwhile, the processor 120 may include heterogeneous processing units.For example, the processor 120 may include a processing unit and aneural processing unit (NPU) that operate based on an operating systemsuch as a central processing unit (CPU).

The NPU may be a processor dedicated for neural computation, and mayinclude a plurality of processing elements. One-way shift or two-wayshift of data is possible between adjacent processing or computationelements of the NPU.

Each of the computation elements may basically include a multiplier andan Arithmetic Logic Unit (ALU), and the ALU may include at least oneadder. The computation element may perform arithmetic operations using amultiplier and an ALU. However, the present disclosure is not limitedthereto, and may be formed in any other structures as long as it canperform functions such as arithmetic operations, shifts, etc. Inaddition, each of the computation elements may include a register forstoring data.

If the processor 120 includes a processing unit and an NPU that operatebased on an operating system, the processing unit may identify the typeof an input image by inputting the input image to the second artificialintelligence model, and the NPU may process the input image by inputtingthe input image to one of a plurality of first artificial intelligencemodels based on the identified type.

However, the present disclosure is not limited thereto, and theprocessor 120 may further include a processing unit dedicated for signalprocessing such as a Digital Signal Processor (DSP), a GraphicsProcessing Unit (GPU), etc. In this case, the processing unit dedicatedfor signal processing such as a DSP may identify the type of an inputimage by inputting the input image to the second artificial intelligencemodel, and the NPU may process the input image by inputting the inputimage to one of the plurality of first artificial intelligence models.

Meanwhile, functions related to artificial intelligence according to anembodiment are operated through the processor 120 and the memory 110.

The processor 120 may consist of one or a plurality of processors. Inthis case, the one or a plurality of processors may be a general purposeprocessor such as a CPU, an AP, a DSP, etc., a processor dedicated forgraphics such as a GPU and a Vision Processing Unit (VPU), or aprocessor dedicated for artificial intelligence such as an NPU.

One or a plurality of processors may control to process input dataaccording to a predefined operation rule or an artificial intelligencemodel stored in the memory 110. Alternatively, based on one or aplurality of processors being a processor dedicated for artificialintelligence, the processor dedicated for artificial intelligence may bedesigned in a hardware structure specialized for processing a specificartificial intelligence model. The predefined operation rule or theartificial intelligence model is characterized by being made throughtraining.

Here, being made through training means that as the artificialintelligence model is trained using a plurality of pieces of learningdata by a learning algorithm, a predefined operation rule or anartificial intelligence model set to perform a desired feature (orpurpose) is made. Such training may be conducted in an apparatus itselfwhere artificial intelligence is performed or through a separate serverand/or system. The examples of the learning algorithm include supervisedlearning, unsupervised learning, semi-supervised learning orreinforcement learning, but are not limited thereto.

The artificial intelligence model may comprise or consist of a pluralityof neural network layers. Each of the plurality of neural network layersmay have a plurality of weighted values and may perform a neural networkoperation through computation between a computation result of theprevious layers and the plurality of weighted values. The plurality ofweighted values of the plurality of neural network layers may beoptimized by a learning result of an artificial intelligence model. Forexample, the plurality of weighted values may be updated to reduce orminimize a loss value or a cost value obtained in an artificialintelligence model during a learning process.

The artificial neural network may include a Deep Neural Network (DNN).For example, the artificial neural network may include a ConvolutionalNeural Network (CNN), Deep Neural Network (DNN), a Recurrent NeuralNetwork (RNN), a Restricted Boltzmann Machine (RBM), a Deep BeliefNetwork (DBN), a Bidirectional Recurrent Deep Neural Network (BRDNN), aGenerative Adversarial Network (GAN), a Deep Q-Network, etc., but is notlimited thereto.

Meanwhile, the electronic apparatus 100 may further include a displayand a communication interface.

The display may be implemented in various types of displays such as amonitor, a projector, a Liquid Crystal Display (LCD), an Organic LightEmitting Diodes (OLED) display, a Plasma Display Panel (PDP), a microLight Emitting Diode (LED), Laser Display, virtual reality (VR) device,an augmented reality (AR) device, Glasses, etc. The display may alsoinclude a driving circuit, a backlight unit, etc. which can beimplemented in the form of a Thin-film transistor (TFT), such as, ana-si TFT, a low temperature poly silicon (LTPS) TFT, an organic TFT(OTFT), etc. Meanwhile, the display may be implemented as a touch screendisplay, a flexible display, three-dimensional (3D) display, etc.combined with a touch sensor.

The communication interface may be configured to perform communicationwith various types of external devices according to various types ofcommunication methods. The communication interface may include aWireless Fidelity (WiFi) module, a Bluetooth module, an infraredcommunication module, a wireless communication module, etc. Here, eachcommunication module may be implemented in the form of at least onehardware chip.

The processor may perform communication with various external devicesusing the communication interface. Here, the external devices mayinclude a server, a Bluetooth earphone device, a display device, etc.

The WiFi module and the Bluetooth module may perform communication in aWiFi method and a Bluetooth method, respectively. In the case of usingthe WiFi module or the Bluetooth module, various connection informationsuch as a service set identifier (SSID), a session key, etc. may betransmitted and received first, and communication may be established byusing this, and then various kinds of information may be transmitted andreceived.

The infrared communication module may perform communication according toan Infrared Data Association (IrDA) technology using infrared lightwhich lies between visible light and millimeter waves for short-distancewireless data transmission.

The wireless communication module may include at least one communicationchip that performs communication according to various wirelesscommunication standards such as Zigbee, 3rd generation (3G), 3rdgeneration partnership project (3GPP), long term evolution (LTE), LTEAdvanced (LTE-A), 4th generation (4G), and 5th generation (5G), otherthan the above-described communication methods.

In addition, the communication interface may include at least one of alocal area network (LAN) module, an Ethernet module, or a wiredcommunication module performing communication by using a pair cable, acoaxial cable, an optical fiber cable, or the like.

The communication interface may further include an input/outputinterface. The input/output interface may be one of High DefinitionMultimedia Interface (HDMI), Mobile High-Definition Link (MHL),Universal Serial Bus (USB), Display Port (DP), Thunderbolt, VideoGraphics Array (VGA) port, RGB port, D-subminiature (D-SUB), and DigitalVisual Interface (DVI).

The input/output interface may input/output at least one of an audiosignal or a video signal.

According to an embodiment, the input/output interface may include aport for inputting/outputting only an audio signal and a port forinputting/outputting only a video signal separately, or may beimplemented as one port that inputs/outputs both an audio signal and avideo signal.

As described above, the electronic apparatus may identify the type of aninput image using an artificial intelligence model, and improve theprocessing performance of the input image by adaptively processing theinput image using another artificial intelligence model corresponding tothe identified type.

Hereinafter, the operation of the processor 120 will be described ingreater detail using various drawings.

FIG. 3 is a view provided to explain an operation of a processor 120according to an embodiment.

An image enhancing (processing) module 310 may be performed by a CPU ora DSP. A prediction module 320 may be performed by one of a CPU, a DSPor an NPU.

First of all, the prediction module 320 may identify the type of theinput image by inputting the input image to the second artificialintelligence model. In this case, the prediction module 320 maydown-scale the input image, and identify the type by inputting thedown-scaled image to the second artificial intelligence model. Theprediction module 320 may determine whether to perform down-scalingbased on at least one of the resolution, capacity and extension of theinput image. Here, the input image and the second artificialintelligence model may be provided from the memory 110.

An NPU 330 may perform a neural network operation by inputting the inputimage to the second artificial intelligence model, and output the imagetype as a result.

An image processing module 310 may perform pre-processing with respectto the input image. For example, the image processing module 310 mayscale the resolution of the input image to correspond to a plurality offirst artificial intelligence models. Alternatively, the imageprocessing module 310 may change the data type of the input image tocorresponding to the input format of the NPU.

The image processing module 310 may process the input image by inputtingthe input image to one of the plurality of first artificial intelligencemodels. Here, the input image and one of the plurality of firstartificial intelligence models may be provided from the memory 110.

The NPU may perform Neural Network (NN)-processing by inputting theinput image to one of the plurality of first artificial intelligencemodels, and output the image-processed input image as a result.

The image processing module 310 may perform post-processing with respectto the image-processed input image. For example, the image processingmodule 310 may filter and output the image-processed input image.Alternatively, the image processing module 310 may change the data typeso that the image-processed input image corresponds to the input formatof the module after the image processing module 310.

When information regarding the type of the input image is received, thememory 110 may provide the NPU 330 with the first artificialintelligence model corresponding to the type of the input image fromamong the plurality of first artificial intelligence models.

Meanwhile, FIG. 3 illustrates an embedded system performing a neuralnetwork operation in software using the NPU 330, but the presentdisclosure is not limited thereto. For example, the embedded system maybe implemented in hardware through a digital circuit design such asRegister-Transfer Level (RTL). When using a single artificialintelligence model, it is more advantageous to implement the embeddedsystem in RTL in terms of power or speed.

However, according to an embodiment, a plurality of first artificialintelligence models and the second artificial intelligence model arerequired, and the plurality of first artificial intelligence models andthe second artificial intelligence model may have different structuresor may have different weights even if there have the same structure. Inother words, in the case of using various artificial intelligence modelsalternately, it would be more advantageous to perform an operation insoftware using the NPU 330 than using RTL. In particular, the NPU 330may have better performance and efficiency than other processing unitsdue to an accelerator specialized for a neural network operation.

FIGS. 4A and 4B are views provided to explain an operation of theprocessor 120 according to an image type according to an embodiment. Forconvenience of explanation, FIGS. 4A and 4B describe that upscaling ofan input image is performed, and the type of the image corresponds toone of none or blurring.

As illustrated in FIG. 4A, if the type of the input image is identifiedas none, the processor 120 may process the input image by inputting theinput image to the first artificial intelligence model A (NN A).

Alternatively, as illustrated in FIG. 4B, if the type of the input imageis identified as blurring, the processor 120 may process the input imageby inputting the input image to the first artificial intelligence modelB (NN B). In this case, the result output by the first artificialintelligence model B (NN B) may be in a state in which blurring isminimized. For example, the input image may have less blurring when itis input to the first artificial intelligence model B (NN B) than whenit is input to the first artificial intelligence model A (NN A).

FIGS. 5A to 5C are views provided to explain a learning method of aplurality of first artificial intelligence models and the secondartificial intelligence model according to an embodiment. Forconvenience of explanation, FIGS. 5A to 5C describe that upscaling ofthe input image is performed, a server learns the plurality of firstartificial intelligence models and the second artificial intelligencemodel, and the input image corresponds to one of two types.

First of all, FIG. 5A is a view illustrating that a plurality of sampleimages (Input A-Input N) are input to one 510 of the plurality of firstartificial intelligence models to obtain a plurality of up-scaled sampleimages (Output A-Output N), and the type of each of the plurality ofup-scaled sample images (Output A-Output N) is identified. Here, one 510of the plurality of first artificial intelligence models may be anartificial intelligence model obtained by learning through an artificialintelligence algorithm to perform up-scaling regardless of the type andhereinafter, for convenience of explanation, one 510 of the plurality offirst artificial intelligence models will be described as a basicartificial intelligence model 510.

For example, the basic artificial intelligence model 510 may be obtainedby learning a relationship between a plurality of sample images and anoriginal image corresponding to each sample image through an artificialintelligence algorithm. Here, the plurality of sample images may be in astate in which there is no distinction between types.

A server may identify the type of the input image by comparing theoutput of the basic artificial intelligence model 510 regarding aplurality of sample images with the corresponding original images. Forexample, the server may identify that blurring has occurred by comparingthe first sample image with the corresponding original image.

The server may store a plurality of sample images and an image typecorresponding to each sample image.

As illustrated in FIG. 5B, the server may train the second artificialintelligence model 520 using sample images which are classifiedaccording to types. For example, the server may input sample imageswhich belong to the first type (type 1) and the second type (type 2) tothe second artificial intelligence model 520, and train the secondartificial intelligence model 520 so that the second artificialintelligence model 520 outputs an identifier corresponding to each type.However, the server is not limited thereto, and may input sample imageswhich belong to the first type (type 1) to the second artificialintelligence model 520, input sample images which belong to the secondtype (type 2) to the second artificial intelligence model 520, and trainthe second artificial intelligence model 520 so that an identifiercorresponding to the second type is output.

After the training of the second artificial intelligence model 520 iscompleted, when the input image is input to the second artificialintelligence model 520, the second artificial intelligence model 520 mayoutput an identifier corresponding to the type of the input image.

As illustrated in FIG. 5C, the server may train one of the firstartificial intelligence models for each type. For example, the firstartificial intelligence model A (NN A) corresponding to the first typemay be obtained by learning a relationship between sample images (InputA-Input M) belonging to the first type and an original imagecorresponding to each sample image through an artificial intelligencealgorithm, and the first artificial intelligence model B (NN B)corresponding to the second type may be obtained by learning arelationship between sample images (Input a-Input n) belonging to thesecond type and an original image corresponding to each sample imagethrough an artificial intelligence algorithm.

In FIG. 5C, it is described that both the first artificial intelligencemodel A (NN A) and the first artificial intelligence model B (NN B) aretrained, but the present disclosure is not limited thereto. For example,if the first type is none, the learning of the first artificialintelligence model A (NN A) may be omitted and the basic artificialintelligence model 510 may be used. In this case, only the firstartificial intelligence model (NN B) regarding the second type may betrained. Alternatively, if the first type is none, the basic artificialintelligence model 510 may be trained and updated with respect to arelationship between sample images belonging to the first type and anoriginal image corresponding to each sample image through an artificialintelligence algorithm. The training of the first artificialintelligence model B (NN B) may be performed in the same manner.

Through the above-described method, a plurality of first artificialintelligence models and the second artificial intelligence model may beobtained.

FIG. 6 is a view provided to explain real time computation according toan embodiment.

As described above, the second artificial intelligence model may beimplemented in a simpler structure than the plurality of firstartificial intelligence models, and the input image may also bedown-scaled and input to the second artificial intelligence model.

In particular, FIG. 6 illustrates a processing speed when the secondartificial intelligence model is an Inception model or a Mobilenetmodel.

For example, if the second artificial intelligence model is implementedas an Inception model and an NPU operating at 840 MHz and having aspecific structure is used, it is possible to identify the type of theinput image at a speed of 154 frames per second (fps).

In general, in the case of the electronic apparatus 100, especially anelectronic apparatus having a display, several frames of an image may bestored in the memory of the final display module. This is because adecoder or internal image quality processing modules may not be able toprocess one frame within a specific period (e.g., 30 hertz (Hz), 60 Hz,120 Hz). In other words, in order to ensure continuity of the screen,several frames image-processed in advance are stored in a buffer, andthe electronic apparatus may display them according to a specificperiod. Therefore, when the type of the input image is identified at thespeed as illustrated in FIG. 6, real-time performance may be secured.

FIG. 7 is a view provided to explain an update of a plurality of firstartificial intelligence models and a second artificial intelligencemodel according to an embodiment.

First of all, a server 200 may be an apparatus for training theplurality of first artificial intelligence models and the secondartificial intelligence model. The server 200 may train the plurality offirst artificial intelligence models and the second artificialintelligence model using the methods as illustrated in FIGS. 5A to 5C.

The server 200 may provide the electronic apparatus 100 with theplurality of first artificial intelligence models and the secondartificial intelligence model of which training is completed. Theelectronic apparatus 100 may receive the plurality of first artificialintelligence models and the second artificial intelligence model ofwhich training is completed from the server 200 via the communicationinterface.

As illustrated in FIG. 2, the electronic apparatus 100 may perform imageprocessing with respect to the input image. The electronic apparatus 100may obtain a weighted value of a plurality of types related to the inputimage in the process of identifying the type of the input image. Inaddition, if at least one of the weighted values of the plurality typesis equal to or greater than a threshold value, the electronic apparatus100 may store the input image and the at least one weighted value.

As such, the electronic apparatus 100 may accumulatively store the inputimage of which weighted value is equal to or greater than a thresholdvalue from among input images which are input sequentially andinformation regarding the weighted value.

The electronic apparatus 100 may provide the sever 200 with informationregarding the input images and the weighted value corresponding to eachinput image which is stored accumulatively at at least one of:predetermined time intervals or when the electronic apparatus 100 isturned on.

The server 200 may update the plurality of first artificial intelligencemodels and the second artificial intelligence model based on theinformation received from the electronic apparatus 100, and provide theelectronic apparatus 100 with the updated plurality of first artificialintelligence models and second artificial intelligence model.

As additional learning is performed with the input image of whichweighted value is equal to or greater than a threshold value asdescribed above, the second artificial intelligence model maydistinguish each type more clearly, and the plurality of firstartificial intelligence models may be more specialized in imageprocessing for each type.

FIG. 8 is a flowchart provided to explain a controlling method of anelectronic apparatus according to an embodiment.

First of all, the type of an input image may be identified by inputtingthe input image to the second artificial intelligence model trained toidentify the type of the image by predicting the image processing resultby a plurality of first artificial intelligence models trained toperform different image processing. (Operation S810). The input imagemay be processed by inputting the input image to one of the plurality offirst artificial intelligence models based on the identified type(Operation S820).

Here, the step of down-scaling the resolution of the input image isfurther included, and the step of identifying (Operation S810) mayinclude identifying the type of the input image by inputting thedown-scaled image to the second artificial intelligence model.

The step of down-scaling may include, if the size of the input image isgreater than a predetermined size, down-scaling the input image.

Meanwhile, the second artificial intelligence model may be an artificialintelligence model obtained by learning a relationship between aplurality of sample images and the type corresponding to each sampleimage through an artificial intelligence algorithm, and each of theplurality of first artificial intelligence models may be an artificialintelligence model obtained by learning a relationship between thesample image of the type corresponding to each of the plurality of firstartificial intelligence modules from among the plurality of sampleimages and the original image corresponding to the sample image throughan artificial intelligence algorithm.

Here, the type corresponding to each sample image may be obtained byprocessing each of the plurality of sample images by inputting each ofthe plurality of sample images to one of the plurality of firstartificial intelligence models and may be obtained based on the type ofthe side effect of each of the plurality of processed sample images.

The type of the side effect may include at least one of none, jagging,blurring, aliasing, or noise boosting.

Meanwhile, the step of processing may include at least one of up-scalingprocessing, noise removing processing or detail enhancement processingof the input image.

In addition, the step of processing (Operation S820) may includeobtaining a weighted value regarding a plurality of types related to theinput image by inputting the input image to the second artificialintelligence model and identifying the type having the largest weightedvalue from among the plurality of types as the type of the input image.

Here, if at least one of the weighted values regarding the plurality oftypes is equal to or greater than a threshold value, the step of storingthe input image and the at least one weighted value may be furtherincluded.

Meanwhile, the operation of identifying (Operation S810) may includeidentifying the type of the input image by inputting the input image tothe second artificial intelligence model by the processing unit of theelectronic apparatus which may operate based on an operating system, andthe operation of processing (Operation S820) may include processing theinput image by inputting the input image to one of the plurality offirst artificial intelligence models based on the identified type by theNPU of the electronic apparatus.

According to the various embodiments of the present disclosure, theelectronic apparatus may identify the type of an input image using anartificial intelligence model, and improve processing performance of theinput image by processing the input image using another artificialintelligence model corresponding to the identified type.

The above-described various embodiments of the disclosure may beimplemented as software including instructions that may be stored inmachine-readable storage media (e.g., a transitory media, such as, asignal wave, or a non-transitory computer readable medium), which can beread by a machine (e.g., a computers). The machine may refer to anapparatus that calls instructions stored in a storage medium, and thatcan operate according to the called instructions. The apparatus mayinclude an electronic apparatus (e.g., an electronic apparatus (A))according to the embodiments described in the disclosure. When aninstruction is executed by a processor, the processor may perform afunction corresponding to the instruction by itself, or by using othercomponents under its control. The instruction may include a code that isgenerated or executed by a compiler or an interpreter. The storagemedium that is readable by machine may be provided in the form of anon-transitory storage medium. Here, the term ‘non-transitory’ onlymeans that a storage medium does not include signals, and is tangible,but does not indicate whether the storage medium is non-volatile orvolatile (data is stored in the storage medium semi-permanently ortemporarily).

In addition, according to an embodiment of the disclosure, the methodaccording to the various embodiments described above may be providedwhile being included in a computer program product. A computer programproduct refers to a product, and it can be traded between a seller and abuyer. The computer program product can be distributed on-line in theform of a storage medium that is readable by machines (e.g., a compactdisc read only memory (CD-ROM)), or through an application store (e.g.,Google Play™ store). In the case of on-line distribution, at least aportion of the computer program product may be stored in a storagemedium such as the server of the manufacturer, the server of theapplication store, and the memory of the relay server at leasttemporarily, or may be generated temporarily.

In addition, according to an embodiment of the disclosure, theaforementioned various embodiments of the disclosure may be implementedin a computer or a recording medium that can be read by an electronicdevice (e.g., a computer) by using software, hardware or a combinationthereof In some cases, the embodiments described in this disclosure maybe implemented as a processor itself. Meanwhile, according toimplementation by software, the embodiments such as procedures andfunctions described in this disclosure may be implemented as separatesoftware modules. Each of the software modules may perform one or morefunctions and operations described in this disclosure.

Meanwhile, computer instructions for performing processing operations ofdevices according to the aforementioned various embodiments of thedisclosure may be stored in a non-transitory computer-readable medium.When computer instructions stored in such a non-transitorycomputer-readable medium are executed by the processor of a specificdevice, processing operations at devices according to the aforementionedvarious embodiments are made to be performed by the specific device. Anon-transitory computer-readable medium refers to a medium that isphysical/tangible. A non-transitory computer-readable medium may storedata semi-permanently, and may be readable by machines. Thenon-transitory computer-readable medium may be different from aregister, a cache, and a memory. As specific examples of anon-transitory computer-readable medium, there may be a compact disc(CD), a digital versatile disc (DVD), a hard disc, a blue-ray disc, auniversal serial bus (USB), a memory card, a read-only memory (ROM) andthe like.

Also, each of the components according to the aforementioned variousembodiments (e.g., a module or a program) may comprise or consist of asingular object or a plurality of objects. In addition, among theaforementioned corresponding sub components, some sub components may beomitted, or other sub components may be further included in the variousembodiments. Generally or additionally, some components (e.g., a moduleor a program) may be integrated as an object, and perform the functionsthat were performed by each of the components before integrationidentically or in a similar manner. Operations performed by a module, aprogram, or other components according to the various embodiments may beexecuted sequentially, in parallel, repetitively, or heuristically. Or,at least some of the operations may be executed in a different order, oromitted, or other operations may be added.

While preferred embodiments of the disclosure have been shown anddescribed, the disclosure is not limited to the aforementioned specificembodiments, and it is apparent that various modifications can be madeby those having ordinary skill in the art, without departing from thegist of the disclosure as claimed by the appended claims, and suchmodifications are not to be interpreted independently from the technicalidea or prospect of the disclosure.

What is claimed is:
 1. An electronic apparatus comprising: a memory thatstores first information regarding a plurality of first artificialintelligence models trained to perform image processing differently fromeach other and second information regarding a second artificialintelligence model trained to identify a type of an image by predictinga processing result of the image by each of the plurality of firstartificial intelligence models; and a processor configured to: identifya type of an input image by inputting the input image to the secondartificial intelligence model stored in the memory, and process theinput image by inputting the input image to one of the plurality offirst intelligence models stored in the memory based on the identifiedtype.
 2. The electronic apparatus as claimed in claim 1, wherein theprocessor is further configured to down-scale a resolution of the inputimage and identify a type of the input image by inputting thedown-scaled input image to the second artificial intelligence model. 3.The electronic apparatus as claimed in claim 2, wherein the processor isfurther configured to, based on a size of the input image being equal toor greater than a predetermined size, down-scale the input image andidentify a type of the input image by inputting the down-scaled inputimage to the second artificial intelligence model.
 4. The electronicapparatus as claimed in claim 1, wherein the second artificialintelligence model is an artificial intelligence model obtained bytraining a relationship between a plurality of sample images and a typecorresponding to each sample image through an artificial intelligencealgorithm, and wherein each of the plurality of first artificialintelligence models is an artificial intelligence model obtained bytraining a relationship between a sample image of a type correspondingto each of the plurality of first artificial intelligence models fromamong the plurality of sample images and an original image correspondingto the sample image through an artificial intelligence algorithm.
 5. Theelectronic apparatus as claimed in claim 4, wherein the typecorresponding to each sample image is obtained by processing each of theplurality of sample images by inputting each of the plurality of sampleimages to one of the plurality of first artificial intelligence models,and obtained based on a type of a side effect of each of the processedplurality of sample images.
 6. The electronic apparatus as claimed inclaim 5, wherein the type of a side effect includes at least one ofnone, jagging, blurring, aliasing, or noise boosting.
 7. The electronicapparatus as claimed in claim 1, wherein the processing the input imageincludes at least one of up-scaling processing, noise removingprocessing or detail enhancement processing.
 8. The electronic apparatusas claimed in claim 1, wherein the processor is further configured toobtain a weighted value regarding a plurality of types related to theinput image by inputting the input image to the second artificialintelligence model, and identify a type having a largest weighted valuefrom among the plurality of types as a type of the input image.
 9. Theelectronic apparatus as claimed in claim 8, wherein the processor isfurther configured to, based on at least one of weighted valuesregarding the plurality of types being equal to or greater than athreshold value, store the input image and the at least one weightedvalue in the memory.
 10. The electronic apparatus as claimed in claim 1,wherein the processor includes a central processing unit (CPU) and aNeural Processing Unit (NPU) which operate based on an Operating System,wherein the CPU is configured to identify the type of the input image byinputting the input image to the second artificial intelligence model,and wherein the NPU is configured to process the input image byinputting the input image to one of the plurality of first artificialintelligence models based on the identified type.
 11. A method ofcontrolling an electronic apparatus, the method comprising: identifyinga type of an input image by inputting the input image to a secondartificial intelligence model trained to identify a type of an image bypredicting an image processing result by each of a plurality of firstartificial intelligence models trained to perform image processingdifferently from each other; and processing the input image by inputtingthe input image to one of the plurality of first artificial intelligencemodels based on the identified type.
 12. The method as claimed in claim11, further comprising: down-scaling a resolution of the input image,wherein the identifying comprises identifying a type of the input imageby inputting the down-scaled input image to the second artificialintelligence model.
 13. The method as claimed in claim 12, wherein thedown-scaling comprises, based on a size of the input image being equalto or greater than a predetermined size, down-scaling the input image.14. The method as claimed in claim 11, wherein the second artificialintelligence model is an artificial intelligence model obtained bytraining a relationship between a plurality of sample images and a typecorresponding to each sample image through an artificial intelligencealgorithm, and wherein each of the plurality of first artificialintelligence models is an artificial intelligence model obtained bytraining a relationship between a sample image of a type correspondingto each of the plurality of first artificial intelligence models fromamong the plurality of sample images and an original image correspondingto the sample image through an artificial intelligence algorithm. 15.The method as claimed in claim 14, wherein the type corresponding toeach sample image is obtained by processing each of the plurality ofsample images by inputting each of the plurality of sample images to oneof the plurality of first artificial intelligence models, and obtainedbased on a type of a side effect of each of the processed plurality ofsample images.
 16. The method as claimed in claim 11, furthercomprising: based on at least one of weighted values regarding theplurality of types being equal to or greater than a threshold value,storing the input image and the at least one weighted value.
 17. Themethod as claimed in claim 11, wherein the identifying comprisesidentifying a type of the input image by inputting the input image tothe second artificial intelligence model by a central processing unit(CPU) of the electronic apparatus which operates based on an OperatingSystem, and wherein the processing comprises processing the input imageby inputting the input image to one of the plurality of first artificialintelligence models based on the identified type by a Neural ProcessingUnit (NPU) of the electronic apparatus.
 18. The method as claimed inclaim 11, wherein the identifying comprises identifying a type of theinput image by inputting the input image to the second artificialintelligence model by a central processing unit (CPU) of the electronicapparatus which operates based on an Operating System, and wherein theprocessing comprises processing the input image by inputting the inputimage to one of the plurality of first artificial intelligence modelsbased on the identified type by a Neural Processing Unit (NPU) of theelectronic apparatus.
 19. A computer-implemented method of training aneural network for image processing comprising: collecting a set ofdigital sample images from a database; inputting the collected set ofdigital facial images into a plurality of first neural network models,so as to obtain a plurality of outputs, wherein the plurality of firstneural network models are trained to perform image processingdifferently from each other; training, in a second neural network model,relationships between the set of digital sample images and a side effecttype corresponding to each digital sample image using the obtainedplurality of outputs, wherein the side effect type corresponds to a sideeffect of processing by a respective one of the plurality of firstneural network models.
 20. The computer-implemented method of claim 19,further comprising: identifying a side effect type of an input digitalimage by inputting the input digital image to the second artificialintelligence model, and processing the input digital image by inputtingthe input digital image to one of the plurality of first intelligencemodels that corresponds to the identified defect type.